In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.
In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.
Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.
Visualize the German Traffic Signs Dataset. This is open ended, some suggestions include: plotting traffic signs images, plotting the count of each sign, etc. Be creative!
The pickled data is a dictionary with 4 key/value pairs:
# Load pickled data
import pickle
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.image as pimg
import cv2
import tensorflow as tf
from sklearn.model_selection import train_test_split
from datetime import timedelta
from time import time
%matplotlib inline
# TODO: fill this in based on where you saved the training and testing data
training_file = 'train.p'
testing_file = 'test.p'
with open(training_file, mode='rb') as f:
train = pickle.load(f)
with open(testing_file, mode='rb') as f:
test = pickle.load(f)
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']
# First, let's check what traffic classes exist to images from training file
# And we know there are 0 - 42 classes of traffic signs exist in training
np.bincount(y_train).nonzero()[0]
# Next, let's check what traffic classes exist to images from testing file
# Also we know there are 0 - 42 classes of traffic signs exist in testing
np.bincount(y_test).nonzero()[0]
# Briefly check the traffic class distribution in training
np.bincount(y_train)
# Briefly check the traffic class distribution in testing
np.bincount(y_test)
sign_names = pd.read_csv('signnames.csv')
classes = np.bincount(y_train).nonzero()[0]
fig = plt.figure(figsize=(15,6))
ax = fig.add_subplot(111)
x = classes
yTrain = np.bincount(y_train)
yTest = np.bincount(y_test)
bar_train = ax.bar(x, yTrain, width=0.5, color='b')
bar_test = ax.bar(x+0.5, yTest, width=0.5, color='r')
ax.set_xlabel('Traffic Sign Classes')
ax.set_ylabel('Sample Counts')
ax.set_xticks(x+0.5)
ax.set_xticklabels(sign_names.SignName.astype(str) + ' ' + sign_names.ClassId.astype(str), rotation = 90)
ax.legend((bar_train[0], bar_test[0]), ('training samples', 'testing samples'))
plt.show()
### To start off let's do a basic data summary.
# TODO: number of training examples
n_train = y_train.shape[0]
# TODO: number of testing examples
n_test = y_test.shape[0]
# TODO: what's the shape of an image?
image_shape = X_train[0].shape
# TODO: how many classes are in the dataset
# check y_test has the same traffic classes as y_train
n_classes = 0
if np.array_equal(np.bincount(y_train).nonzero()[0], np.bincount(y_test).nonzero()[0]):
n_classes = np.bincount(y_test).nonzero()[0].size
min_train_cls = np.argmin(yTrain)
max_train_cls = np.argmax(yTrain)
min_test_cls = np.argmin(yTest)
max_test_cls = np.argmax(yTest)
print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
print("----------------------")
print("Training set: minimum class = {}, count = {}".format(min_train_cls, yTrain[min_train_cls]))
print("Training set: maximum class = {}, count = {}".format(max_train_cls, yTrain[max_train_cls]))
print("Testing set: minimum class = {}, count = {}".format(min_test_cls, yTest[min_test_cls]))
print("Testing set: maximum class = {}, count = {}".format(max_test_cls, yTest[max_test_cls]))
### Data exploration visualization goes here.
### Feel free to use as many code cells as needed.
# Now we know that each class of training or testing traffic signs has got some images,
# to plot all images to each class is not feasible as in this notebook, so instead I'll
# plot random-k images from each traffic sign class, and set k=1 below!
# Helper function to plot images side-by-side with title
def plot_gallery(images, titles, h, w, n_row=3, n_col=5):
"""Helper function to plot a gallery of portraits"""
plt.figure(figsize=(2.0 * n_col, 2.4 * n_row))
plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)
llen = 21
for i in range(min(n_row * n_col, len(images))):
plt.subplot(n_row, n_col, i + 1)
plt.imshow(images[i], cmap=plt.cm.gray)
title_i = titles[i]
if len(title_i) >= llen:
title_i = titles[i][:llen] + '\n' + titles[i][llen:]
plt.title(title_i, size=12)
plt.xticks(())
plt.yticks(())
# It is not feasible to plot all traffic signs to each Class in either train or test set,
# so only plot the 1st sign of a Class from training images as visualization
# find 1st occurring Indices of unique traffic signs
u, indices = np.unique(y_train, return_index=True)
# u is array of unique ClassIds in ascending order, align with sign_names['SignName']
plot_gallery(X_train[indices], sign_names['SignName'], 32, 32, 5, 9)
# plot one traffic sign for example
plt.imshow(X_train[0])
X_train[0].shape
Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.
There are various aspects to consider when thinking about this problem:
Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.
Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.
### Preprocess the data here.
### Feel free to use as many code cells as needed.
# Choose not to do grayscale on original images here, due to:
# - Image dimension is 32x32x3, not much concern in converting to 32x32x1 in saving computation
# - It is harder to normalize a grayscaled image, since grayed_pixel_val = kind_of_mean(R,G,B),
# its pixel value distribution is no more uniform & min/max is different from image to image
# - Like to let CNN learn the color scheme of traffic sign to boost result consider that there
# are no different signs of a same class be different only in colors (where grayscale helps)
# Then TWO primary preprocesses are implemented here:
# 1. Image normaliztion as (R/G/B-128.)/128, to achieve 0 mean & normal distribution (-1., 1.)
X_train_norm = (X_train - 128.)/128
X_test_norm = (X_test - 128.)/128
# 2. Class Label OneHot Encoding
y_test_ohot = np.eye(n_classes)[y_test]
# N.B. Final performance report will be using:
# X_test_norm, y_test_ohot
# vs.
# X_test, y_test
# N.B. Not do OneHot on global training labels as below now
# y_train_ohot = np.eye(n_classes)[y_train]
#
# Instead OneHot is done on training & validation set from train_test_split later
Describe the techniques used to preprocess the data.
Answer:
### Generate data additional (if you want to!)
### and split the data into training/validation/testing sets here.
### Feel free to use as many code cells as needed.
print("Prior split Training set: minimum class = {}, count = {}".format(min_train_cls, yTrain[min_train_cls]))
print("Prior split Training set: maximum class = {}, count = {}".format(max_train_cls, yTrain[max_train_cls]))
# Knowing that minimum sample count of a class is 210 & max sample count of a class is 2250 in training samples
# We must not split training & validation set by random percentage! Must use stratify instead to retain ratios!
# Randomly spare ~20% images cross Class for validation, leave testing set untouched until final report.
X_t, X_v, y_t, y_v = train_test_split(X_train_norm, y_train, test_size=0.2, random_state=101, stratify=y_train)
# To confirm splitted train_set (X_t, y_t) & validation_set (X_v, y_v) both have all the Classes
# and retain a similar cross-class samples distribution as prior, let's do visualize :)
fig = plt.figure(figsize=(15,5))
ax = fig.add_subplot(111)
x = classes
yT = np.bincount(y_t)
yV = np.bincount(y_v)
bar_t = ax.bar(x, yT, width=0.5, color='b')
bar_v = ax.bar(x+0.5, yV, width=0.5, color='y')
ax.set_xlabel('Traffic Sign Classes')
ax.set_ylabel('Sample Counts')
#ax.set_xticks(x+0.5)
#ax.set_xticklabels(sign_names.SignName.astype(str) + ' ' + sign_names.ClassId.astype(str), rotation = 90)
ax.legend((bar_t[0], bar_v[0]), ('trainning samples', 'validation samples'))
plt.show()
min_t_cls = np.argmin(yT)
max_t_cls = np.argmax(yT)
min_v_cls = np.argmin(yV)
max_v_cls = np.argmax(yV)
print("Post split Training set: minimum class = {}, count = {}".format(min_t_cls, yT[min_t_cls]))
print("Post split Training set: maximum class = {}, count = {}".format(max_t_cls, yT[max_t_cls]))
print("Post split Validation set: minimum class = {}, count = {}".format(min_v_cls, yV[min_v_cls]))
print("Post split Validation set: maximum class = {}, count = {}".format(max_v_cls, yV[max_v_cls]))
# Now do OneHot on training and validation labels
y_to = np.eye(n_classes)[y_t]
y_vo = np.eye(n_classes)[y_v]
Describe how you set up the training, validation and testing data for your model. If you generated additional data, why?
Answer:
### Define your architecture here.
### Feel free to use as many code cells as needed.
# Network architecture (Preliminary choice: 2 ConvNets + 2 FullConnected NNets. Main constraints: Computational Power!)
# Shape of tensors passing along the graph is annotated in (); basic layers formation is annotated in []
# Images (,32,32,3)=> CNN1[3x5x5x16] =(,16,16,16)=> CNN2[16x5x5x32] =(,8,8,32)=> FC1[2048x128] ==> FC2[128x43] => Logits
# N.B.
# - FC2 nodo ReLU
# - FC1 todo dropout (optional)
# Global Architecture Parameters Definition:
# Use 5x5 kernel at both Convolutional Layers
ck_size = 5
# Depth (output channels) of CNN1 and CNN2
c1depth = 16
c2depth = 32
# Input size to FC1 is known 2048 (flat of 8x8x32)
# Output from FC2 is known 43 (num of classes)
# Size between fully connected layers: FC1 and FC2
fc_size = 128
# Definitions of Images
img_size = 32
imgdepth = 3
imgshape = (img_size, img_size, imgdepth)
# Helper functions
def init_weight(shape):
# be sure to use small stddev, as model starts with unsure
return tf.Variable(tf.truncated_normal(shape, stddev=0.001))
def init_bias(size):
return tf.Variable(tf.constant(0.001, shape=[size]))
# N.B. first dimension of input tensor is arbitrary, it is the number of samples determined at feed
# Parameters -
# input : input tensor
# out_depth : output depth or channels
# k_size : ConvNet kernel patch size, e.g. 5 for 5x5
# maxpool : use maxpooling to down sampling output ?
# relu : use ReLU (Rectified Linear Unit) activation ?
def newCN(input, out_depth, k_size = 5, maxpool = True, relu = True):
# ConvNet patch weight shapes, according to TensorFlow API
in_depth = input.get_shape()[-1:].num_elements()
wshape = [k_size, k_size, in_depth, out_depth]
# Build ConvNet patch weight with initial value
weight = init_weight(wshape)
# Build ConvNet patch biases with initial value
bias = init_bias(out_depth)
# Create the TensorFlow operation for convolution.
# N.B. the strides are set to 1 in all dimensions.
# This stride produce tensor w. same Width,Height.
# We use max_pool to down-sample output if needed.
conv = tf.nn.conv2d(input = input,
filter = weight,
strides = [1, 1, 1, 1],
padding = 'SAME')
# Add bias to output channels
conv += bias
if maxpool:
# only 2x2 max-pooling for now
conv = tf.nn.max_pool( value = conv,
ksize = [1, 2, 2, 1],
strides = [1, 2, 2, 1],
padding = 'SAME')
if relu:
# only ReLU to add non-linear activation for now
conv = tf.nn.relu(conv)
# return the result conv layer and parameters
# return conv, weight, bias
return conv
# N.B. first dimension of input tensor is arbitrary, it is the number of samples determined at feed
# Parameters -
# input : input tensor, do flatten in function if it is not already
# out_nn : output number of neurons
# relu : use ReLU (Rectified Linear Unit) activation ? (it should be False at last FC layer)
# dropout : use dropout as regularization to avoid overfit ?
# N.B. practice:
# - Output FC layer: no ReLU, no dropout
# - Hidden FC layer: w/ ReLU, dropout optional for overfit tuning
def newFC(input, out_nn, relu = False, dropout = False):
input_features = input.get_shape()[1:].num_elements()
if input.get_shape()[1:] != input.get_shape()[-1:]:
# reshape input to flatten
flat_input = tf.reshape(input, [-1, input_features])
else:
flat_input = input
weight = init_weight([input_features, out_nn])
bias = init_bias(out_nn)
# Y = X*W + b
fc = tf.matmul(flat_input, weight) + bias
if relu:
fc = tf.nn.relu(fc)
if dropout:
# fc = tf.nn.dropout(fc, keep_prob = 0.5)
# Do NOT use hard coded keep_prob
# It MUST be feed differently: training (0.5) -OR- valiation (1.0) !!
# keep_prob = tf.placeholder(tf.float32)
fc = tf.nn.dropout(fc, keep_prob)
return fc
# Placeholder for input images tensor
X_image = tf.placeholder(tf.float32, shape=[None, img_size, img_size, imgdepth])
# Placeholder for true class labels tensor
y_class = tf.placeholder(tf.float32, shape=[None, n_classes])
# True Class IDs of OneHot tensors
# N.B. we have global true class labels y_t, y_v, y_test,
# but those are not lined up with a tensor, so use this:
y_true_class = tf.argmax(y_class, dimension = 1)
keep_prob = tf.placeholder(tf.float32)
# CNN1: 5x5 kernel, 3 input color channels, 16 output filters; use 2x2 max_pooling and ReLU
cnn1 = newCN(input = X_image, out_depth = c1depth, k_size = 5, maxpool = True, relu = True)
# CNN2: 5x5 kernel, 16 input channels, 32 output channels; use 2x2 max_pooling and ReLU
cnn2 = newCN(input = cnn1, out_depth = c2depth, k_size = 5, maxpool = True, relu = True)
# FC1 - hidden layer, [-1, 8, 8, 32] :=> FC1 :=> [-1, 128]
fc1 = newFC(input = cnn2, out_nn = fc_size, relu = True, dropout = True)
# FC2 - output layer, [-1, 128] :=> FC2 :=> [-1, 43]
fc2 = newFC(input = fc1, out_nn = n_classes, relu = False)
What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.
Answer:
Finding the optimal architecture of a ConvNet for a given task remains mainly empirical, I'd like to investigate and research further in area such as GridCV search in network architecture.
### Training Evaluation Primitives
learn_rate = 0.001
# Logits is One-Hot output of FC2
logits = fc2
# Prediction in softmax probabilities
y_pred = tf.nn.softmax(logits)
# Prediction in Class IDs
y_pred_class = tf.argmax(y_pred, dimension = 1)
# Come up Loss function of supervised classification
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits = logits, labels = y_class)
# Loss is the batch -or- global mean of cross_entropy
# It will be used by optimizer to minimize @ training
loss = tf.reduce_mean(cross_entropy)
# Optimizer
optimizer = tf.train.AdamOptimizer(learning_rate = learn_rate).minimize(loss)
# Good prediction tensor from TWO equal length comparison tensors,
# e.g. batch -or- global
good_pred = tf.equal(y_pred_class, y_true_class)
# Predictor Accuracy, pertain to the length of the tensor,
# e.g. batch scope accuracy
accuracy = tf.reduce_mean(tf.cast(good_pred, tf.float32))
### Train your model here.
### Feel free to use as many code cells as needed.
epochs = 100
# batch size
bsize = 128
# display progress every batch
# N.B. last batch may not have batch_size samples
# Total number of training batches (per epoch)
batches_t = -(- y_t.shape[0] // bsize)
# Total number of validation batches (if batching)
batches_v = -(- y_v.shape[0] // bsize)
# Total number of testing batches (if batching)
batches_T = -(- y_test.shape[0] // bsize)
# N.B. Training, Validation, Testing set:
# X: normalized Y: OneHot
# Training: X_t, y_to
# Validation: X_v, y_vo
# Testing: X_test_norm, y_test_ohot
# Simple version to get ith batch from a dataset,
# Index i: [0, total_batches)
def next_batch(i, X, y):
total_batches = -(- y.shape[0] // bsize)
if i < 0:
print("Batch Index is outboundry!")
return
elif i < total_batches-1:
X_b = X[i*bsize: (i+1)*bsize]
y_b = y[i*bsize: (i+1)*bsize]
elif i == total_batches-1:
X_b = X[i*bsize:]
y_b = y[i*bsize:]
else:
print("Batch Index is outboundry!")
return
return X_b, y_b
start = time()
# Data structures to records training and validation measures
# So we can visualize learning curve. Use python list for now
# These are per epoch
Epoch = [] # Epoch No
Acc_t = [] # Accuracy of training
Acc_v = [] # Accuracy of validation
Acc_T = [] # Accuracy of Testing
Losst = [] # Loss of training
Lossv = [] # Loss of validation
LossT = [] # Loss of Testing
# Start TensorFlow Session
# Used Session manager as below initially, but encountered Session restoration at later steps
# for extra prediction analysis. Those will be resolved with checkpoint mgmt at a later time!
# with tf.Session() as sess:
sess = tf.Session()
sess.run(tf.initialize_all_variables())
Epoch.clear()
Acc_t.clear()
Acc_v.clear()
Acc_T.clear()
Losst.clear()
Lossv.clear()
LossT.clear()
for epoch_i in range(epochs):
# Training
Epoch.append(epoch_i)
for batch_i in range(batches_t):
X_train_b, y_train_b = next_batch(batch_i, X_t, y_to)
feed_dict_train = {X_image: X_train_b, y_class: y_train_b, keep_prob: 0.5}
sess.run(optimizer, feed_dict=feed_dict_train)
"""
# Print training status at each batch_step
# N.B. IF to print per batch status, then uncomment these lines.
batch_step = 200
if batch_i % batch_step == 0:
feed_dict_train[keep_prob] = 1.0
batch_acc_t, batch_loss_t = sess.run([accuracy, loss], feed_dict=feed_dict_train)
training_msg = "Epoch: {0:>2}, Batch: {1:>6}, Training Accuracy: {2:>6.1%}, Training Loss: {3:.8f}"
print(training_msg.format(epoch_i, epoch_i * batches_t + batch_i, batch_acc_t, batch_loss_t))
"""
# Measurement
# At end of each epoch, we get full status of training, validation (and testing) measures:
feed_dict_t = {X_image: X_train_b, y_class: y_train_b, keep_prob: 1.0}
acc_t, loss_t = sess.run([accuracy, loss], feed_dict=feed_dict_t)
training_msg = "Epach: {0:>3}, Training Accuracy: {1:>6.1%}, Training Loss: {2:.8f}"
print(training_msg.format(epoch_i, acc_t, loss_t))
Acc_t.append(acc_t)
Losst.append(loss_t)
feed_dict_v = {X_image: X_v, y_class: y_vo, keep_prob: 1.0}
acc_v, loss_v = sess.run([accuracy, loss], feed_dict=feed_dict_v)
validation_msg = "Epach: {0:>3}, Validation Accuracy: {1:>6.1%}, Validation Loss: {2:.8f}"
print(validation_msg.format(epoch_i, acc_v, loss_v))
Acc_v.append(acc_v)
Lossv.append(loss_v)
feed_dict_T = {X_image: X_test_norm, y_class: y_test_ohot, keep_prob: 1.0}
acc_T, loss_T = sess.run([accuracy, loss], feed_dict=feed_dict_T)
testing_msg = "Epach: {0:>3}, Testing Accuracy: {1:>6.1%}, Testing Loss: {2:.8f}"
print(testing_msg.format(epoch_i, acc_T, loss_T))
Acc_T.append(acc_T)
LossT.append(loss_T)
plt.figure(figsize=(12,6))
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.title('Learning Curve - Accuracy per Epoch')
plt.plot(Epoch, Acc_t, 'bs-', label='training')
plt.plot(Epoch, Acc_v, 'g^-', label='validation')
plt.plot(Epoch, Acc_T, 'ro-', label='testing')
plt.legend(loc='lower right')
plt.show()
plt.figure(figsize=(12,6))
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Learning Curve - Loss per Epoch')
plt.plot(Epoch, Losst, 'bs-', label='training')
plt.plot(Epoch, Lossv, 'g^-', label='validation')
plt.plot(Epoch, LossT, 'ro-', label='testing')
plt.legend(loc='upper right')
plt.show()
#### Validation curves converged with training curves at HIGH plateau (99.0~100.0%) in Accuracy & Very LOW (< 0.04) in LOSS. It means:
#### Testing curves not converged with training/validation curves, but it plateaued HIGH (~95%) in Accuracy & flattened LOW (< ~0.4) in LOSS
How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)
Answer:
batch_stop_training_report.txt in submission, where overfitting without dropout is observablebatch_stop_training_report.txt in submission), but outcome is not very intuitive as visualization, so I detoured to plot learning curve in the final report.¶What approach did you take in coming up with a solution to this problem?
Answer:
Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.
You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.
Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
# Use imperfect (no smooth may lose detail) resize from cv2:
# img = cv2.resize(img, (32, 32))
# Choose new images from various labels, and one type not yet in label space:
#
# New Images files captured # Projected Class ID (Human Perf.) from model's label space if ANY
newImages = ['20kmh_CHN.png', # class 0 - very few samples in training set
'Yield_CA.png', # class 13 - lots of samples in training set
'STOP_CA.png', # class 14 - enough samples in training set
'KeepRightLED_CHN.png', # class 38 - lots of samples in training set
'NoUTurn_CA.png'] # None - no this sample class in training
newTruth = [0, 13, 14, 38, None]
newImgCnt = len(newImages)
newImgX = np.empty((newImgCnt, 32, 32, 3))
for i in range(newImgCnt):
img = pimg.imread(newImages[i])
img = cv2.resize(img, (32,32))
newImgX[i,:,:,:] = img[:,:,0:3]
# Visualize all 5 new Traffic Sign Images to be classified
# Some loss of detail obversed
plot_gallery(newImgX, newImages, 32, 32, 1, 5)
# Now run the same preprocessing that classifier requires on these new Images
# Same normalization as in training: (R/G/B-128.)/128 => 0 mean & normal distribution between (-1., 1.)
# X_train_norm = (X_train - 128.) / 128
# !!! but now all newImgX images are already normalized but differently in range (0., 1.) vs. (-1., 1.)
# This is due to we read them in PNG format!
# Then it is safe to do a PATCH Normalization as:
newImgX_norm = (newImgX - 0.5) * 2
plot_gallery(newImgX_norm, newImages, 32, 32, 1, 5)
Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It would be helpful to plot the images in the notebook.
Answer: Five candidate images new to the test are:
### Run the predictions here.
### Feel free to use as many code cells as needed.
print("The Human Truth labels of these to be classified Traffic Signs are:\n{}\n".format(newTruth))
new_pred = np.zeros(shape = newImgCnt, dtype=np.int)
new_pred = sess.run(y_pred_class, feed_dict = {X_image: newImgX_norm, keep_prob: 1.0})
print("The Classifier Predicted labels of these new set Traffic Signs are:\n{}\n".format(new_pred))
print(sign_names['SignName'][new_pred])
print("\n")
print("The Predicition Accuracy on these new set of Traffic Signs are: {0:>6.2%}\n"
.format((new_pred == np.array(newTruth)).sum() / newImgCnt))
print("OK let's visualize - the new traffic signs to be classified, and then what they are classified to:")
plot_gallery(newImgX, newImages, 32, 32, 1, 5)
plot_gallery(newImgX, list(sign_names['SignName'][new_pred]), 32, 32, 1, 5)
# Interestingly we have got these TWO Errors:
# Classified
# 1. 20kmh_CHN (China) [Projected Class: 0] == wrongly to ==> [Class: 13] Yield
# 3. No U-Turn (California) [Projected Class: NA] == no choice ==> [Class: 19] Dangerous curve to the left
Is your model able to perform equally well on captured pictures or a live camera stream when compared to testing on the dataset?
Answer: Not really. It only achieves 60.00% Accuracy on this new test set, 2/5 new samples are misclassified.
"20kmh (China)" is wrongly classified to "Yield (Germany)", let's plot 9 random Yield sign from German training set been used:¶cls13 = np.where(y_train==13)[0]
cls13_r9 = np.random.choice(cls13, size=9, replace=False)
plot_gallery(X_train[cls13_r9], [sign_names['SignName'][13]]*9, 32, 32, 2, 9)
"No U-Turn(California)" is in-no-choice classified to "Dangerous curve to the left", let's plot 9 random target sign in training:¶cls19 = np.where(y_train==19)[0]
cls19_r9 = np.random.choice(cls19, size=9, replace=False)
plot_gallery(X_train[cls19_r9], [sign_names['SignName'][19]]*9, 32, 32, 2, 9)
target traffic signs above, I have come to an earlier observation:¶#### "20kmh(China)" -- classified --> "Yield(Germany)" is totaly nonsense from visual, need an interesting insight!
"20kmh(China)" sign has got different Yellow background as German speed limit signsChinese characters at the bottom, that may be countable noiseYield vs. others, will do some softmax probabilities analysis next.great in memorizing seen patterns vs. potentially poor in never seen patterns, have more data to training is always one of the best solutions.#### "No U-Turn(California)" ---> "Dangerous curve to the left" make some sense, in that there is a common "Left Turn" partial pattern exist in their center BLACK MARK!
"No U-Turn" is out of our model's label space (0-42), its mis-classification is expected, there is no right choice."out of label" can contribute to big Systematic Error in supervised learning!"Global Traffic Sign Classifier", then we need to feed it with Global Signs.### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.
# Okay let's take an error examples above to visualize its softmax probability from classification output:
# Classified
# 1. 20kmh_CHN (China) [Projected Class: 0] == wrongly to ==> [Class: 13] Yield
# 2. No U-Turn (California) [Projected Class: NA] == no choice ==> [Class: 19] Dangerous curve to the left
# Prediction in softmax probabilities
# y_pred = tf.nn.softmax(logits)
top5 = tf.nn.top_k(y_pred, k=5, sorted=True, name=None)
top_prob, top5_ind = sess.run(top5, feed_dict = {X_image: newImgX_norm, keep_prob: 1.0})
# top5_ind is the top5 Class IDs of softmax probabilities in descending order to each of these 5 new test images
print(top5_ind)
20kmh (China) is classified as Top5 likely classes [13 17 12 38 25] in descending likelyhood, let's visualize them side-by-side!¶nImg_Indice = []
nName_Likes = []
# Build collection of likelihood Images (select random ONE from each likely Class)
for i in range(top5_ind.shape[1]):
clsId = top5_ind[0][i]
clsImgInd = np.where(y_train == clsId)[0]
Img_1Like = np.random.choice(clsImgInd, size=1, replace=False)
nImg_Indice.append(Img_1Like[0])
like_Images = np.array(nImg_Indice)
nName_Likes.append(sign_names['SignName'][clsId])
plt.imshow(newImgX[0])
plot_gallery(X_train[like_Images], nName_Likes, 32, 32, 1, 5)
print("Visualization of the mis-classifing 20kmh_CHN:")
print("Plotted below is Top5 softmax probability Images in descending likelihood order (left is the most likely one)")
"Yield" with Probability 9.99999642e-01 (dominant)"No entry" with Probability 3.02646725e-07"Priority road" with Probability 2.78513934e-10"Keep right" with Probability 3.63664177e-12"Road work" with Probability 8.11566743e-13"Yield" is thought to be the closest one, but we noticed that they all seem to have Yellow color around the Red border and White color area inside.¶No U-Turn (CAL) is classified as Top5 likely classes [19 12 21 10 11] in descending likelyhood, let's visualize them side-by-side!¶nImg_Indice = []
nName_Likes = []
# Build collection of likelihood Images (select random ONE from each likely Class)
for i in range(top5_ind.shape[1]):
clsId = top5_ind[4][i]
clsImgInd = np.where(y_train == clsId)[0]
Img_1Like = np.random.choice(clsImgInd, size=1, replace=False)
nImg_Indice.append(Img_1Like[0])
like_Images = np.array(nImg_Indice)
nName_Likes.append(sign_names['SignName'][clsId])
plt.imshow(newImgX[4])
plot_gallery(X_train[like_Images], nName_Likes, 32, 32, 1, 5)
print("Visualization of the mis-classifing No U-Turn_CA:")
print("Plotted below is Top5 softmax probability Images in descending likelihood order (left is the most likely one)")
"Dangerous curve to the left" with Probability 9.46826458e-01 (dominant)"Priority road" with Probability 5.31702675e-02"Double curve" with Probability 2.98974282e-06"No passing for vechiles over 3.5 metric tons" with Probability 1.88694088e-07"Right-of-way at the next intersection" with Probability 1.31326814e-07Dangerous curve to the left is thought to be the closest one, due to the common left turn half in black exist. This pattern of Red border around White inner area with certain Black curvly mark inside the center is quite common in all top5 candidates, except for the none intuitive 2nd candidate - Priority road.¶# Top5 probabilities of 5 new samples.
print(top5_prob)
Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)
Answer:
# close tf.Session object. It is commented out to avoid accidental close(), which will lose
# training model prior to a checkpoint save(training) / restore(predict) being put in place
# sess.close()
If necessary, provide documentation for how an interface was built for your model to load and classify newly-acquired images.
Answer:
Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.